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ABSTRACT 

Information relative to What const itutes, quality 
vocational programs is necessary as a basis for program evaluation. 
M098 (1968) has defihed program evaluation as the process of 
attributing differences between actual and comparative outcomes to 
program characteristics under different conditions of student 
characteristics and intervening influences , and judging the value of 
program" characteristics. The definition has two important 
qualities — evaluation must be comparative and differences must be 
attributable t<> program characteristics or interaction of programs 
and student characteristics. Few reports have directly addressed the 
concept, of "quality 11 or what' constitutes quality in vocational 
education. Quality, howeuer, must be assured as quantity of 
vocational education has increased. Three factors are suggested as 
determining quality: scope, management by objective, and bench marks. 
Common criteria used in judging quality of the vocational education 
process are a stated set of objectives, quality of stufenJt commitment 
and motivation, quality of faculty, facilities and equipment, quality 
of program or curriculum, and feedback mechanisms Cor evaluation. 
Process or product criteria must 'be chosen for program evaluation. A 
viable measure would be. the degree the .program increases graduate 
employability, or educe s. measurable benefits of achievement, and 
improves retention rates. Outcomes other than employment should be 
included as non traditional criteria for program evaluation. (TLB) 
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TOWARDS QUALITY VOCATIONAL EDUCATION PROGRAMS: 
IMPLICATIONS FOR PROGRAM EVALUATION 



By: 

N. Alan Sheppard 
Associate Professor of Education 
Division of Vocational and Technical Education 
Virginia Polytechnic Institute and State University 

Concern about th^ quality r-f programs in education transcends vocational 
education and the field of pro ial education. The concept of "quality" 

is an illusive a. .a ill-defined c. In the way che terra seems t be 

commonly used "quality" is that h is in the eye of the beholder. 

Considerations of quality invoLve considerations about an end product 
or resulting set of conditicns, a set of specified characteristics, are 
standards upon which judgements can.be based.' In some cases, the matter of 
quality is a relatively simple one. However,* in vocational education We 
have no widely agreed upon specifications and we have no clearly defined 
un f I orni standards . 

ft appears frequently that the widespread differences of view about 
quality in vocational education programs are differences about the process 
ur>n<\ to measure and/or document such quality. ? 

A number of evaluative c . forts have been completed that have attempted 
to document the effectiveness of "ocational education programs. Often this 
of feeti veness is determined by criterion such as the percentage of students 
placed In the occupation for which they were trained, occupational adep^ness, 
salaries of employees, etc. 

Even without the authorization to evaluate and/or assess vocational 
education programs as authorised in PL 94-482 (section 523 (b) of PL 94-482), 
a basic rationale of this paper is that program evaluation for quality and 
effectiveness is important because it provides evidence about Jjie relative 
merits programs, thus enabling educators to make more realistic decisions 
about: program operation's, development and/or modifications, The key, of course 




is improved racionality of decision making in relation to soma quality 
criterion. 

With regards to criteria I or quality assessment, thLs paper will 
address and note the vaviations, sometimes very great variations, in programs 
in different states. Fcr more than fifty years, federal statutes have im- 
used federal policy and regulations, and a considerable amount of uniformity 
does 4 exist. But differences in population, economic patterns and educational 
development within the states h<tve .had counteracting effects leading to 
substantial differences in the i r vocational education programs; thusly, 
leading one to conclude that indeed "quality vocational education prop, rams" 
is in the eye of the beholder. 

The purpose of this paper is to present information relative to what 
constitutes quality vocational programs as a basis for program evaluation. 
Further this paper will address t lie importance of program evaluation, 
criteria for evaluating quality vocational education programs, the need for 
non-t i nlitional criteria and the need for new instrumentation and evaluative 
stra t vi\ i e^ in vocational education. 



THE IMPORTANCE OF PROGRAM EVALUATION: 
DEFINITION , OBJECTIVES AND ROLE 

I ho term evaluation has its roots in the words evaluer and valer e. The , 

*~ u 

Freiuli word e valuer means to estimate, and the latin word valere means to be 

strong. From valere comes the word value , meaning the worth or quality of 
something. Evaluer implies a process whereas valere implies a product. Thus, 
evaluation is thought of as a process pr product of evaluating. 

This background helps to explain why definitions of evaluation include 
concepts of process, estimation and value. To illustrate the use of these 
concepts a few definitions have been gleaned from educational literature and 
are presented here: 

Evaluation is th^ process of determining the extent changes in 
knowledges, interests, understandings, attitudes and skills were 
accomplished. f ^ 



ERLC 



-3- 



Evaluation is the process of assessing the degree to wnlch one 
is arhieving his objectives. 

Evaluation is a process of determining L he extent to which the 
educational objectives of a program have been reached at the 
end of a particular cchuaLional activity. 

[Evaluation is the systematic attempt to gat'ier evidence regarding 
(lunges in rtudent behavior that accompany educational experiences* 

Evaluation is the process of ascertaining or judging the value or 
amount of something by careful appraisal. f 

These definitions of evaluation consider evaluation a process (or attempt) 
to determine (or ascertain, assess, or judge) the degree (or extent, value or 
amount) of progress (or change, effectiveness or achievement) toward pre- 
determined objectives (or behaviors oc something). Evaluation, then, is an 
active process of trying to find arswers to specific question. 

it is apparent that program evaluation should achieve three major 
objectives; First , to determine whether program objectives have been reached. 
Second, to provide data for planning and/or a rational basis for decision 
making. Third , to determine whether programs meet criteria for reimbursement. 
Ideally, priorities ought to be in tint order. Initially a given situation 
n;iy require that the latter be the first priority. 

While all vocational, technical and practical arts educators have some 
comept of program evaluation, the literature indicates that the meanings 
Ik- Id are quite varied. Figure 1 entitled "Major ' ".opponents of the Evaluative 
^System", is intended to introduce the concept as it is used in this paper. 

The model, which was proposed by Moss (1967), starts with students, 

each of whom differ with respect to characteristics which affect their ability 

to learn at the time they^ enter the program to be evaluated. Students differ, 

for example in relevant aptitudes, achievement, motivation, health, etc. 

which alone and in interactions, create variation in "readiness" for the 

* 

program. 

The program the students enter has characteristics which provide them 
with educational experiences. It is these characteristics that are to be 
evaluated. Students are exposed to selected content, which has been organ- 
ized in specific ways, which is presented in certain manners,. gpd to which 
the students are encouraged to respond in particular ways, all under the 
guidance or management of an instructor with certain characteristics. These 
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Figure 1 
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"transactions" (Stake, 1967) Lako place under particular physical and psycho- 
social environmental conditions. 

Fn addition to the influences of the specific program to be evaluated, 
students are inevitably affected by other experiences and conditions in the 
environment, which occur outside of the program, but whose effects might be 
mistaken for outcomes of the program. These experiences can take place at 
any time pfter the student enters the program, and before the program out- 
comes are measured. For example, students might rake a variety of other 
courses which differentially alter their ability to learn the concent of the 
program to be evaluated; increases in dependents or extra-curr icul.i experience 
on a part-time summer or aftei -school job could change motivation; economLc 
conditions could alter the availability of particular ki^ds of jobs after 
graduation; military service could result in greatly enhanced occupational 
ski 1 Is , etc . 

The Intera :tion of student characteristics, program characteristics, and 
nth "V intervening influences produce actual outcomes. These outcomes 
consist of student or ex-student behavLors, and the effect of those behaviors 
on the school, the community, the economy, society, etc., and other direct 
consequences ->l the program lur tea* hers, administrative pa tern«-, ether 
students, etc. 

finally, the eVn 1,i Ttive system contains one or more set of i mparative 
out tomes. These outcomes are anticipated, expected, hoped ior results of the 
program, or they may he the actual outcomes of a different program, or the 
outcomes of the same program at different points in time. Tn all cases, they 
provide the comparative standard by which the relative merits of i given 
program will be judged. 

['ho components of the evaluative system can be utilized to create a 

mor 1 formal definition of program evaluation, as follows: 

Program evaluation is the process of attributing differences 
between actual and comparative outcomes to program character- 
istics, under different conditions of student characteristics 
and other intervening influences, and making a judgment about i 
the value of the program characteristics. The process is con- \ 
ducted for the purpose of making more rational decisions 
about programs (Moss, 1%8) 

Note that the definition has two important qualities. First, evaluation 
must be comparative. To report actual, outcot »»s provides only a description 
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of what happened. Evaluation requires making a judgement, which in turn 
necessitates comparing outcomes with some other set of expected or actual 
outcomes. 

Second, evaluation requires that differences in the outcomes compared 

must be attri butable to program characteristics or the interaction of program 
— p 

and student characteristics. Comparing outcomes which do not reflect actual 
"differences in programs, but whijh are clue, for example, to differences in 
students, or to other relevant influences, would be completely misleading. 

CHARACTERISTICS OF QUALITY PROGRAMS IN 
VOCATIONAL EDUCATION: A BRIEF SYNOPSIS 

A report of the National Planning Association (1972) suggests that 
successful or quality vocational education programs must take into account 
both the educational and the economic dimensions. Another way to look at 
quality vocational education programs is by functions — service to a 
.technically oriented society and service to individuals in society, (OF, 1972). 

Further, this same report I icluded a three-part checklist — program, 
ocoti'iny, and student — to measure program characteristics of qmllty 
vocational education programs. ih* 1 report however did not indicate how 
those quality characteristics were derived. 

ha^onais (1974) disrussed the identification of the most surcessful 
vocational education programs in California community colleges using the 
holphl method. The characteristics of the successful programs were grouped 
in the areas of student, curriculum, instruction, administration, and advisory 
committee* This report pointed out the reliability of pee opinion in 
rating the various criteria. 

A group of anonymous panel members was utilized with a three-phase 
polling technique. First, panel members were asked to identify the five 
most successful programs, using their own judgment about the definition of 
successful . 

The second round consisted of asking the panel members to review a 
composite list of successful programs and select three most successful 
programs instead of five. 

In the third round the reasons for identification of successful programs 
were rated by panel members in terms of their relative importance to program 

8 



sin These reasons were combined with opinions obtained elsewhere and 

t ht n incorporated into a questionnaire used to collect data on seh Led 
programs for the empirical part of the study. 

A statistical discussion pertaining to reliability of the i>clphl panel 
was included in the report. 'I his seems to have been the major focus of the 
study in relation to quality characteristics observed. 

Successful Programs that were indentified included: 

1. Large programs. 

2. Programs that enrolled females in greater proportion than males. 
'3. Programs that were regulated through State licensing procedures. 

4 . Programs that screened applicants . 

5. Programs that did not offer remedial courses. 

The report further included a statistical discussion relating to the 
establishment of these criteria. 

On an empirical level the study identified the multivariate nature of 
siu frss in vocational education programs. Twenty items with potential for 
predicting program success were Identified. Of particular interest is the 
fart that the 20 variables distribute themselves among five major topical 
groups: student y cur ricul um, faculty , management , and trusteeshi p . A 
study by Wallace (1975) which deals with the search for quality in vocational 
education programs provides a list of components for "quality" vocational- 
technical programs. This list of quality components is used in a Sel f - 
Checklist evaluation form devised as an alternative to the State on-site 
•instrument. The report is not conclusive as to the superiority of this 
ins I rui ent . 

A preponderance of studies and/or documents further noted that program 
evaluation which attempts to assess the "quality" of the program is dependent 
upon comparing the actual observed outcome of a program with some standard. 
Three performance standards frequently referred to are: (1) arbitrarily 
fixed standards (2) standards based or. norm groups and (3) comparative 
standards. 

In summary, there are hut a few reports that directly address the" 
concept of "quality" or what constitutes quality In vocational education 
programs . 
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TOWARDS QUALITY PROGRAMS 

Quality must be assured as quantity of vocational education has increased. 
Many new individuals and institutions are involved in the decision and oper- 
ational processes of vocational education. Each must have essential criteria 
available for guidance. 

What are the factors which determine quality in vocational education 

programs? Among these factors are: 

Scope is a factor . Quality in vocational and technical education 
is not an attribute that resides solely in the individual 
program; it relates to the scope of the program available to 
youth. Efforts to develop vocational and technical programs 
within an area or school division must be concerned with a 
scope of available offerings, not only wi>h the facilities, 
equipment, and instructional staff and curriculum within onQ 
program. 

Finally, a vocational education program cannot be considered 
quality unless it makes available access and opportunity in 
keeping with different abilities, interests and needs of all 
youth and adults to be served; this is especially true since 
the 1963 Act, the 1968 Amendments and PL 94 -482 have combined 
to put greater emphasis on meeting the needs of special popula- 
tions heretofore not adequately met, (ethnic minorities, the 
handicapped , disadvantaged, migrant laborers, the incarcerated, 
women , e tc . ) . 

(b) Management by Objective. I also identify management by objecJrive 
as a basis for quality. Perhaps our objectives upon whLch we 
measure our effectiveness have not been that clear. Then, again, 
perhaps our objectives were as clear or clearer thaa'in other 
fields of education. 

(c) Bench Marks . In many states, including Virginia, legislation 
has been passed, referred to as "Standards of Quality" in the 
Commonwealth, to provide an adequate program of vocational 
education for youth enrolled in the public schools. The leg- 
islation is quite spccLfic and provides evaluators a good basis 
for assessing quality programs. 

Specifically, "what are the 'quality dimensions' that separate the good 
from the mediocre vocational programs?" 

There are common criteria utilized In making Judgments about the quality 
of the process of vocational education. The first dimension or criterion 
is a sta;ed set of objectives which actually give direction to the vocational 
erlticitioti program itself, .my set of objectives which does not limit or 
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}»rtM \\u\t* certain acivities i:> opt-rationally meaningless and useless. 

\ second dimension is the quality of students enrolled in r ho vocational 
and technical programs relative to their commitment and motivation. It has 
heen my experience that vocational education is relevant education. Properly 
organized and properly taught, it can serve not only as a means to prepare 
youth for employment but also as a method of education for a significant 
number of youth for whom subject-centered academic education has little 
meaning. 

The occupational goals of youth encourage commitment to a vocational 
education program. Youth can he motivated to achieve in that program if 
it provides for reasonable choice and if the quality of the program earns their 
respect. You will note that judgments made about the quality of students 
enrolled in vocational programs can reflect considerations about both end 
product and process. 

A third dimension of a quality vocational education program is the quality 
of the faculty Consi derat ions of faculty characteristics also are reflective 
of views about both end product and process. Some of the faculty character- 
istics which I feel are basic are: intelligence, integrity, dedication to 
the field of vocational, real world orientation, teaching and occupat i onal 
t (Jinpc Lenc Les . 

A fourth dimension would be facilities and equipment. It is perfectly 
obvious that without access or a growth of physical facilities and equipment, 
wo cannot possibly implement the quality/quantity concents important in 
vocational and technical education. Further, this could contribute to or 
hinder the development of the skills, technical knowledge, work habits, ?nd 
attitudes essential to entrance into employment. 

A fifth dimension is the quality of the vocational and technical education 
program (or curriculum ) itself. An assessment of this component should 
reflect a curriculum that (1) meets individual needs, (2) takes into consid- 
eration the nature of the society which the instructional prog-am will be 
pro-ided and (3) includes an analysis of the occupational area to be taught — 
analysis of the skills, technical knowledge and other competencies necessary 
for success in an occupation. 

A sixth dimension of a quality program is feedback mechanisms which 
are used for evaluative purposes. Systematic use of feedback continues to 
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foeu-, attention on objectives and provides a basis for correction of tnef- 
IVcllve or insufficient means. 

T wish I could feel confident that these six dimensions constitute 
the totality of quality considerations. However, I believe that the existence 
jf qualitative levels in all six dimensions will not provide necessarily a 
"quality" program. One additional essential ingredient is the "climate" 
of a program. Climate seems to be a gestalt, a dimension resulting from 
the interactions among the other dimensions, Without the gestalt, the other 
dimensions seem to lack both potency and effectiveness. Despit these 
observations, we are still left with the basic questions about quality, 
"Whit is it?" "How do you know when you see it?". 

Possessed with a knowledge of quality dimensions, the essential question 
now must be "What should be the indicators or outcomes of a quality vocational 
edti' 'ition program?" Or "What should be the evaluative criteria for quality 
vw it ional educatio* programs?" 



Among the first decisions to be made is whether to use process or 
product criteria. 

Process criteria are druvm from the inputs and transactions that com- 
\>i f',t» the vocational curriculum. They are largely reflective of how the 
vncvit Ion., education program functions, including the various env ironmentai 
elumnts that presumably determine the degree of its success (ie., such 
factors as the curriculum and how its developed, the use of advisory 
ppchanfsms, the equipment used for instructional purposes, the quality of 
tin 1 (acuity, the process of selecting students, and the manner in which 
plafcments are made to name a few). The premise is that by evaluating 
( >n>gicun characteristics, conclusions regarding the viability of the train- 
ing can be inferred, 

Product criteria, on the other hand, consist of the outcomes of in- 
struction — such as student behaviors and what the training does for him. 
They reflect program objectives and program outcomes — both qualitatively 
and quantitatively, According to Moss (1971), "product criteria are very much 
the proof of curriculum pudding". Thusly, product criteria ought to reflect 
how well the program fullfllls its objectives: the extent to which students 



CRITERIA FOR EVALUATING QUALITY 
VOCATIONAL EDUCATION PROGRAMS 




persist, i^w they find jobs appropriate to their training and how thev per- 
form in tlfese jobs. ' ) 

What should fee the i rul iV.ifors r or cr ! tor ia "of * success of vocational 

/ > " / i ' ' * 

programs? . * * * 

Withoftt a doubt", the most common (and time t^tecV) criteria- for 

vw"ii lonal evaluations are: 4 * ^ 



(1) Program completion rates » t i 

(2) Student competence • * 

(3) Cost-efficiency of the program ? O 

(4) ^Placement r>ate .of students, ^ * ; . . 

(5) Employer -empl&yee satisfaction 

in addition, one finds, from a sursjery glar.ee of the literature, other 
criteria including: % 

(1) Programs relationship to job market profile , 

(2) Program success in meeting vocational aspiration of clientele 

(3) Prcgram success in student performance 

(4) Program level of community support 

(5) Degree of student satisfaction * 

(6) Number of students who have successfully maintained *wjob or have 
moved to better employment 

(7) number of students involved in further training 

[ believe very strongly * that evaluation criteria should begin with the 
criteria identified in or related* to the mission established by he local 
education agency. Lvaluative criteria should be structured to permit local 
Individuality of programs rather than applying only a standard instrument 
as a measuring devise. 

ft seems evident to me that a viable measure will, be the degree the 
program increases the employabi I i Uy of graduates, produce measurable benefit 
of achievement, improves retent ion *rates (ie, decreasing d^op out rates and 
its success in meeting the nerds pf students from special/ populations). 

NON-TRADTT1 ONAL OUTCOMES IN MEASURING- 
THE EFFECTIVENESS OF VOCATIONAL EDUCATION 

i 

If the question \*as asked, "Does vocational education do anything for 
students other than train them for employment?" Most people would probably 
agree that it does, but the State and Federal Vocational Education Agencies 
seem *fther unsure, confused of indif f erent . ^Tf true , and this writer con- 
cures, vocational education should recognize the fact — and consider these 
^different outcomes from participation in its programs as new dimensions or 

13 
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non-fraditional criteria for program evaluation. To a considerable extent, 

Confess and the United^ States Office of Education have locked vocational 

eiliH.it ion into a si ngle-produc L criterion outcome concept, hut vocational 

educators and administrators have let it happen. 

Where is the research to really find out? Where is the document .it ion 

of ,vo aLlonal teachers who say their students learn self-confidence, com- 

'mtinir it-ton skills, work attitudes, and how to get along with employers and 

othei employees? Where are the follow-up studies that list other benefits 

"than job placement from vocational education programs? Many stulonls do 

not j;o into the fields for which they were trained, or even into related 

ficld^. Did they wast* their time taking vocational education? Were the 
rrC.. 

vnratnnal education funds used to pay for their training wasted? 

Ihfs j.s one o£ the most consistently ignored gaps in public knowledge 
about vocational education. It is not the fhult of just vocational edu- 
(atiirs,. All educators, and especially educational researchers, should be 
probing the possibilities of improving learning for all students this ugh 
vocat IonuI v education. Many »nd ivldiKiT vocational educators have been con- f 
villi (d lor years that cmp loyme.nt -related, training programs in the sheools 
'improve communication and ui.it hAuat fc basic skills. This could be ron- 
rlu'iively demonstrated or disproved through longitudinal research involving 
protests and post-tests of'ni.!trhci groupa OFF students vocational education 
and 'n non-vdef tional progr WJth^pie Nation's growing concern 

over b isic skills,- -why .'is t Jnd of research npglected? ^ - 

!he neglect is primarily the responsible ty of vocational - education . 
T<vo much emphasis has been placed sqJU3^ on jJb p la cement* . This* is not* 
•hat education is all about, and vocacional educators know ity^^| practice 
the great majority of them are engaged just as much in, preparing students 
for successful lives as other educators are; It .is* in what they say they ^ 1 
are doing, and the Image they have projected of job training as something 
isolated from the rest of education, that they do- a disservice to them- 
selves and their profession. I believe this tc be especially the case with 
reference to assessing the efficancy of vocr nnai education for students 
who are disadvantaged, handicapped, or simply those who come from culturally 
different backgrounds. 
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IMPROVING EVALUATION TECHNIQUES 
IN VOCATIONAL EDUCATION 

I'll is paper connot be com 1 hided without some observations on the ap- 
pru.ii lies currently being used to evaluate vocational education programs., 
flrown(197l); Denton (1973); Medsker (197a); Felstehausen (1973); Pautler 
CI 07^4 ) ; Boelkner (1974); Moss (1968) and others have generally supported 
the following approaches to the -evaluation of occupational education 
programs : 

(!) Formative evaluation 

(2) Sumnufxive evaluation 

(3) Follcw-ups 

(A) Site team evaluations 

(5) Experiments 

(6) Interrupted time series design 

(7) Cost-benefit analysis 

(8) Regression Analysis 

No observations will be made as to the relative merits of these 
approaches; however, it seems quite clear **o this writer that possibly 
iust as important as what technique to use is the timeliness of the ap- 
plication of the technique. In other words, even after the evaluator 
had 1 hosert appropriate product criteria and valid performance indices to 
meisure them, ami after he has decided how to weight each index in order 
to attain an overall measure of cur iculum effectiveness consistent with 
some rationale for vocational education, there still remains the problem 
of determining when to measure the outcomes. One possible choice is to 
measure the outcomes, such as student achievement, after the student has 
completed the curriculum hut is still in school. The alternative, of 
course, is to measure outcomes after the students have been out of school 
for some period *f time. .The validity of the outcome measures, in terms 
of the behavioral goals of vocational education, can be better assured by 
this approach. .Either approach poses some serious pros and cons to consider 
relative to J:he "time" for getting the best evidence that a vocational 
evocation program is effective.. j 

In addition to evalu^on timeliness, if c valuators .are to determine /' 
whether vocational -education, really makes a differencaln terms^of certain 
consequence* of behavior and/or experiences, five' steps must be adhered to: 



j 

(1) The outcomes of vocational education nust be specified 

(2) Means of measuring t he identified outcomes must be established 
( \) Baseline comparisons must be developed 

(4) Data must be collet ted, preferably under experimental conditions 
(if possible) 

(5) They must be appropriately analyzed 

Concluding Comments 

This writer would like to conclude this paper on a very positive note. 
Within ihe past few years, possibly due to increased emphasis on accounta- 
bility and evaluation as mandated by federal, legislation, the technical 

t * \ 

improvements in evaluation have been jreat . To be sure, we still must deal 
with our shortcomings -relative to an operational definition of "quality" 
vocational programs, deciding on what should really be the most consistent 
indices of a quality vocational education program, and deciding on what 
other dimensions constitute the totality of a quality vocational education 
pro gram. 

Consequently, the final judgment must be negative with respect to 
most of the evaluation studies in existence. But the mood is optimistic \ 
as evidenced by the increasing level cf methodological sophisticat ion^rffP 
improvements in professional vocational and technical education graduate 

programs. * 

i 

Although it is clear th.tu t he evaluation of program adequacy is an 
essential part of the assessment of a state program, it also is true... tji>*t 
the value and effectiveness of i program can be determined only with respect 
to ouLcomes. Thus, a program which is effective for objective X may be - 
ineffective for objective Y. A facility, a teaching staff, or a budget 
is appropriate, adequate, or effective onlv with respect to some goal. 
Thus, evaluation of educational program effectiveness is meaningful only 
after it is known whether educational programs served their intended pur- 
pose f> . 

This point is being emphasized strongly in the political and legislative 
arenas. Not only is federal legislation written to emphasize educational 
outcomes for specified groups of students, but education now finds itsel c 
having tp compete at all^e^ernmental levels with other agencies and institu- 
tions for limited huWn and economic resources. Decisions by policy-making 

S s 



bud i rs regarding resource £ fftn^Jj^s are being made with increasing 
frequency o'f evidence of program effectiveness, relevance to social and 
economic conditions, and the degree to which [itograms reflect community, 
state and federal inte^teo^s and concerns. In such an environment, evalu- 
ation methodologies which have been commonly employed in vocational 
education fail to provide the evidence required by policy-making bodies 
and must be replaced by a more effective evaluation methodology If a 
proper case Is to be made for support. 

Despite these observations, we are still left with the basic question 
about a quality vocational education program, "What is it?". 
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